{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "LCL_HW08.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "bDk9r2YOcDc9"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YiVfKn-6tXz8"
      },
      "source": [
        "# **Homework 8 - Anomaly Detection**\n",
        "\n",
        "If there are any questions, please contact ntu-ml-2021spring-ta@googlegroups.com"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w6s3wqonWoxH"
      },
      "source": [
        "# Mounting your gdrive (Optional)\n",
        "By mounting your gdrive, you can save and manage your data and models in your Google drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TW45UmHskzu3",
        "outputId": "a846049a-a30f-47e4-a4bf-47249beda3a1"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fri Aug 20 08:19:03 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 470.57.02    Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   46C    P8    30W / 149W |      0MiB / 11441MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dfqPBGxoWpkC",
        "outputId": "e501fc6e-1d2a-4a19-aa5f-b81aadeac84c"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')\n",
        "import os\n",
        "\n",
        "# your workspace in your drive\n",
        "workspace = 'YOUR_WORKSPACE'\n",
        "\n",
        "\n",
        "try:\n",
        "  os.chdir(os.path.join('/content/gdrive/My Drive/', workspace))\n",
        "except:\n",
        "  os.mkdir(os.path.join('/content/gdrive/My Drive/', workspace))\n",
        "  os.chdir(os.path.join('/content/gdrive/My Drive/', workspace))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bDk9r2YOcDc9"
      },
      "source": [
        "# Set up the environment\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oi12tJMYWi0Q"
      },
      "source": [
        "## Package installation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7LexxyPWWjJB",
        "outputId": "f730e220-6d06-4c91-95bd-8c47a706b970"
      },
      "source": [
        "# Training progress bar\n",
        "!pip install -q qqdm"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  Building wheel for qqdm (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DCgNXSsEWuY7"
      },
      "source": [
        "## Downloading data\n",
        "**Please use the link according to the last digit of your student ID first!**\n",
        "\n",
        "If all download links fail, please follow [here](https://drive.google.com/drive/folders/13T0Pa_WGgQxNkqZk781qhc5T9-zfh19e).\n",
        "\n",
        "* To open the file using your browser, use the link below (replace the id first!):\n",
        "https://drive.google.com/file/d/REPLACE_WITH_ID\n",
        "* e.g. https://drive.google.com/file/d/15XWO-zI-AKW0igfwSydmwSGa8ENb9wCg"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r_wwI4CSWvc2",
        "outputId": "ee6daf9e-81ce-4da6-9aad-3bab1834cb63"
      },
      "source": [
        "\n",
        "!gdown --id '15XWO-zI-AKW0igfwSydmwSGa8ENb9wCg' --output data-bin.tar.gz \n",
        "\n",
        "# Other download links\n",
        "#   Please uncomment the line according to the last digit of your student ID first\n",
        "\n",
        "# 0\n",
        "# !gdown --id '167SejKP7vLB2sbHfQHJii8-WisYoTmLH' --output data-bin.tar.gz \n",
        "\n",
        "# 1\n",
        "# !gdown --id '1BXJaeouaf4Zml2aeNlQfJ_AOcItTWcef' --output data-bin.tar.gz \n",
        "\n",
        "# 2\n",
        "# !gdown --id '1HkBPxhk-9rD0H_cen2YjLXxsvInkToBl' --output data-bin.tar.gz \n",
        "\n",
        "# 3\n",
        "# !gdown --id '1K_WGT8AD8iMsOSMYtK1Gp6vyEcRNCLQM' --output data-bin.tar.gz \n",
        "\n",
        "# 4\n",
        "# !gdown --id '1LGdyDUQA4EPaWTEUVm_upPAEl6qAh91Z' --output data-bin.tar.gz \n",
        "\n",
        "# 5\n",
        "# !gdown --id '1N9wNazaMy4A0UQ6pow5DXfVJ6abaiQxU' --output data-bin.tar.gz \n",
        "\n",
        "# 6\n",
        "# !gdown --id '1PC66MrDw-tnuYN2STauPg2FoJYm3_Yy5' --output data-bin.tar.gz \n",
        "\n",
        "# 7\n",
        "# !gdown --id '1mzy4E06CcBJc0udhPgL4zMhDlWibKbVs' --output data-bin.tar.gz \n",
        "\n",
        "# 8\n",
        "# !gdown --id '1zPbCF7whPv1Xs_2azwe1SUweomgLsVwH' --output data-bin.tar.gz \n",
        "\n",
        "# 9\n",
        "# !gdown --id '1Uc1Y8YYAwj7D0_wd0MeSX3szUiIB1rLU' --output data-bin.tar.gz "
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=15XWO-zI-AKW0igfwSydmwSGa8ENb9wCg\n",
            "To: /content/gdrive/My Drive/YOUR_WORKSPACE/data-bin.tar.gz\n",
            "1.64GB [00:14, 117MB/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1hui5EYoWzB4"
      },
      "source": [
        "## Untar data\n",
        "\n",
        "data-bin contains 2 files\n",
        "```\n",
        "data-bin/\n",
        "├── trainingset.npy\n",
        "├── testingset.npy\n",
        "...\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0K5kmlkuWzhJ",
        "outputId": "0c9e4318-c983-4082-b119-869a6cdecfbf"
      },
      "source": [
        "!tar zxvf data-bin.tar.gz\n",
        "!ls data-bin\n",
        "!ls data-bin\n",
        "!rm data-bin.tar.gz"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "data-bin/\n",
            "data-bin/testingset.npy\n",
            "data-bin/trainingset.npy\n",
            "testingset.npy\ttrainingset.npy\n",
            "testingset.npy\ttrainingset.npy\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HNe7QU7n7cqh"
      },
      "source": [
        "# Import packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jk3qFK_a7k8P"
      },
      "source": [
        "import numpy as np\n",
        "import random\n",
        "import torch\n",
        "\n",
        "from torch.utils.data import DataLoader\n",
        "from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler,\n",
        "                              TensorDataset)\n",
        "import torchvision.transforms as transforms\n",
        "\n",
        "from torch import nn\n",
        "import torch.nn.functional as F\n",
        "from torch.autograd import Variable\n",
        "import torchvision.models as models\n",
        "\n",
        "from torch.optim import Adam, AdamW\n",
        "\n",
        "from sklearn.cluster import MiniBatchKMeans\n",
        "from scipy.cluster.vq import vq, kmeans\n",
        "\n",
        "from qqdm import qqdm, format_str\n",
        "import pandas as pd\n",
        "\n",
        "import pdb  # use pdb.set_trace() to set breakpoints for debugging\n",
        "\n",
        "\n"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6X6fkGPnYyaF"
      },
      "source": [
        "# Loading data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k7Wd4yiUYzAm",
        "outputId": "af6255e8-0c44-491e-f0f1-595d8152abcd"
      },
      "source": [
        "\n",
        "train = np.load('data-bin/trainingset.npy', allow_pickle=True)\n",
        "test = np.load('data-bin/testingset.npy', allow_pickle=True)\n",
        "\n",
        "print(train.shape)\n",
        "print(test.shape)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(140001, 64, 64, 3)\n",
            "(19999, 64, 64, 3)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_flpmj6OYIa6"
      },
      "source": [
        "## Random seed\n",
        "Set the random seed to a certain value for reproducibility."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gb-dgXQYYI2Q"
      },
      "source": [
        "def same_seeds(seed):\n",
        "    # Python built-in random module\n",
        "    random.seed(seed)\n",
        "    # Numpy\n",
        "    np.random.seed(seed)\n",
        "    # Torch\n",
        "    torch.manual_seed(seed)\n",
        "    if torch.cuda.is_available():\n",
        "        torch.cuda.manual_seed(seed)\n",
        "        torch.cuda.manual_seed_all(seed)\n",
        "    torch.backends.cudnn.benchmark = False\n",
        "    torch.backends.cudnn.deterministic = True\n",
        "\n",
        "same_seeds(19530615)"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zR9zC0_Df-CR"
      },
      "source": [
        "# Autoencoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1EbfwRREhA7c"
      },
      "source": [
        "# Models & loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xUUIwFm02qEL"
      },
      "source": [
        "Lecture video：https://www.youtube.com/watch?v=6W8FqUGYyDo&list=PLJV_el3uVTsOK_ZK5L0Iv_EQoL1JefRL4&index=8"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lESPM7C9svYa"
      },
      "source": [
        "fcn_autoencoder and vae are from https://github.com/L1aoXingyu/pytorch-beginner"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GwG2XzgWs_JR"
      },
      "source": [
        "conv_autoencoder is from https://github.com/jellycsc/PyTorch-CIFAR-10-autoencoder/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wi8ds1fugCkR"
      },
      "source": [
        "class fcn_autoencoder(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(fcn_autoencoder, self).__init__()\n",
        "        self.encoder = nn.Sequential(\n",
        "            nn.Linear(64 * 64 * 3, 128),\n",
        "            nn.ReLU(True),\n",
        "            nn.Linear(128, 64),\n",
        "            nn.ReLU(True), \n",
        "            nn.Linear(64, 12), \n",
        "            nn.ReLU(True), \n",
        "            nn.Linear(12, 3))\n",
        "        \n",
        "        self.decoder = nn.Sequential(\n",
        "            nn.Linear(3, 12),\n",
        "            nn.ReLU(True),\n",
        "            nn.Linear(12, 64),\n",
        "            nn.ReLU(True),\n",
        "            nn.Linear(64, 128),\n",
        "            nn.ReLU(True), \n",
        "            nn.Linear(128, 64 * 64 * 3), \n",
        "            nn.Tanh())\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.encoder(x)\n",
        "        x = self.decoder(x)\n",
        "        return x\n",
        "\n",
        "\n",
        "# maybe it can be smaller\n",
        "class conv_autoencoder(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(conv_autoencoder, self).__init__()\n",
        "        self.encoder = nn.Sequential(\n",
        "            nn.Conv2d(3, 12, 4, stride=2, padding=1),         \n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(12, 24, 4, stride=2, padding=1),        \n",
        "            nn.ReLU(),\n",
        "\t\t\t      nn.Conv2d(24, 48, 4, stride=2, padding=1),         \n",
        "            nn.ReLU(),\n",
        "            #nn.Conv2d(48, 96, 4, stride=2, padding=1),   # medium: remove this layer\n",
        "            #nn.ReLU(),\n",
        "        )\n",
        "        self.decoder = nn.Sequential(\n",
        "            #nn.ConvTranspose2d(96, 48, 4, stride=2, padding=1), # medium: remove this layer\n",
        "            #nn.ReLU(),\n",
        "\t\t\t      nn.ConvTranspose2d(48, 24, 4, stride=2, padding=1), \n",
        "            nn.ReLU(),\n",
        "\t\t\t      nn.ConvTranspose2d(24, 12, 4, stride=2, padding=1), \n",
        "            nn.ReLU(),\n",
        "            nn.ConvTranspose2d(12, 3, 4, stride=2, padding=1),\n",
        "            nn.Tanh(),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.encoder(x)\n",
        "        x = self.decoder(x)\n",
        "        return x\n",
        "\n",
        "\n",
        "class VAE(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(VAE, self).__init__()\n",
        "        self.encoder = nn.Sequential(\n",
        "            nn.Conv2d(3, 12, 4, stride=2, padding=1),            \n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(12, 24, 4, stride=2, padding=1),    \n",
        "            nn.ReLU(),\n",
        "        )\n",
        "        self.enc_out_1 = nn.Sequential(\n",
        "            nn.Conv2d(24, 48, 4, stride=2, padding=1),  \n",
        "            nn.ReLU(),\n",
        "        )\n",
        "        self.enc_out_2 = nn.Sequential(\n",
        "            nn.Conv2d(24, 48, 4, stride=2, padding=1),\n",
        "            nn.ReLU(),\n",
        "        )\n",
        "        self.decoder = nn.Sequential(\n",
        "\t\t\t      nn.ConvTranspose2d(48, 24, 4, stride=2, padding=1), \n",
        "            nn.ReLU(),\n",
        "\t\t\t      nn.ConvTranspose2d(24, 12, 4, stride=2, padding=1), \n",
        "            nn.ReLU(),\n",
        "            nn.ConvTranspose2d(12, 3, 4, stride=2, padding=1), \n",
        "            nn.Tanh(),\n",
        "        )\n",
        "\n",
        "    def encode(self, x):\n",
        "        h1 = self.encoder(x)\n",
        "        return self.enc_out_1(h1), self.enc_out_2(h1)\n",
        "\n",
        "    def reparametrize(self, mu, logvar):\n",
        "        std = logvar.mul(0.5).exp_()\n",
        "        if torch.cuda.is_available():\n",
        "            eps = torch.cuda.FloatTensor(std.size()).normal_()\n",
        "        else:\n",
        "            eps = torch.FloatTensor(std.size()).normal_()\n",
        "        eps = Variable(eps)\n",
        "        return eps.mul(std).add_(mu)\n",
        "\n",
        "    def decode(self, z):\n",
        "        return self.decoder(z)\n",
        "\n",
        "    def forward(self, x):\n",
        "        mu, logvar = self.encode(x)\n",
        "        z = self.reparametrize(mu, logvar)\n",
        "        return self.decode(z), mu, logvar\n",
        "\n",
        "\n",
        "def loss_vae(recon_x, x, mu, logvar, criterion):\n",
        "    \"\"\"\n",
        "    recon_x: generating images\n",
        "    x: origin images\n",
        "    mu: latent mean\n",
        "    logvar: latent log variance\n",
        "    \"\"\"\n",
        "    mse = criterion(recon_x, x)  # mse loss\n",
        "    KLD_element = mu.pow(2).add_(logvar.exp()).mul_(-1).add_(1).add_(logvar)\n",
        "    KLD = torch.sum(KLD_element).mul_(-0.5)\n",
        "    return mse + KLD\n",
        "\n",
        "\n",
        "class Resnet(nn.Module):\n",
        "    def __init__(self, fc_hidden1=1024, fc_hidden2=768, drop_p=0.3, CNN_embed_dim=256):\n",
        "        super(Resnet, self).__init__()\n",
        "\n",
        "        self.fc_hidden1, self.fc_hidden2, self.CNN_embed_dim = fc_hidden1, fc_hidden2, CNN_embed_dim\n",
        "\n",
        "        # CNN architechtures\n",
        "        self.ch1, self.ch2, self.ch3, self.ch4 = 16, 32, 64, 128\n",
        "        self.k1, self.k2, self.k3, self.k4 = (5, 5), (3, 3), (3, 3), (3, 3)      # 2d kernal size\n",
        "        self.s1, self.s2, self.s3, self.s4 = (2, 2), (2, 2), (2, 2), (2, 2)      # 2d strides\n",
        "        self.pd1, self.pd2, self.pd3, self.pd4 = (0, 0), (0, 0), (0, 0), (0, 0)  # 2d padding\n",
        "\n",
        "        # encoding components\n",
        "        resnet = models.resnet18(pretrained=False)\n",
        "        modules = list(resnet.children())[:-1]      # delete the last fc layer.\n",
        "        self.resnet = nn.Sequential(*modules)\n",
        "        self.fc1 = nn.Linear(resnet.fc.in_features, self.fc_hidden1)\n",
        "        self.bn1 = nn.BatchNorm1d(self.fc_hidden1, momentum=0.01)\n",
        "        self.fc2 = nn.Linear(self.fc_hidden1, self.fc_hidden2)\n",
        "        self.bn2 = nn.BatchNorm1d(self.fc_hidden2, momentum=0.01)\n",
        "\n",
        "        self.fc3_mu = nn.Linear(self.fc_hidden2, self.CNN_embed_dim)      # output = CNN embedding latent variables\n",
        "\n",
        "        # Sampling vector\n",
        "        self.fc4 = nn.Linear(self.CNN_embed_dim, self.fc_hidden2)\n",
        "        self.fc_bn4 = nn.BatchNorm1d(self.fc_hidden2)\n",
        "        self.fc5 = nn.Linear(self.fc_hidden2, 64 * 4 * 4)\n",
        "        self.fc_bn5 = nn.BatchNorm1d(64 * 4 * 4)\n",
        "        self.relu = nn.ReLU(inplace=True)\n",
        "\n",
        "        # Decoder\n",
        "        self.convTrans6 = nn.Sequential(\n",
        "            nn.ConvTranspose2d(in_channels=64, out_channels=32, kernel_size=self.k4, stride=self.s4,\n",
        "                               padding=self.pd4),\n",
        "            nn.BatchNorm2d(32, momentum=0.01),\n",
        "            nn.ReLU(inplace=True),\n",
        "        )\n",
        "        self.convTrans7 = nn.Sequential(\n",
        "            nn.ConvTranspose2d(in_channels=32, out_channels=8, kernel_size=self.k3, stride=self.s3,\n",
        "                               padding=self.pd3),\n",
        "            nn.BatchNorm2d(8, momentum=0.01),\n",
        "            nn.ReLU(inplace=True),\n",
        "        )\n",
        "\n",
        "        self.convTrans8 = nn.Sequential(\n",
        "            nn.ConvTranspose2d(in_channels=8, out_channels=3, kernel_size=self.k2, stride=self.s2,\n",
        "                               padding=self.pd2),\n",
        "            nn.BatchNorm2d(3, momentum=0.01),\n",
        "            nn.Sigmoid()    # y = (y1, y2, y3) \\in [0 ,1]^3\n",
        "        )\n",
        "\n",
        "\n",
        "    def encode(self, x):\n",
        "        x = self.resnet(x)  # ResNet\n",
        "        x = x.view(x.size(0), -1)  # flatten output of conv\n",
        "\n",
        "        # FC layers\n",
        "        if x.shape[0] > 1:\n",
        "            x = self.bn1(self.fc1(x))\n",
        "        else:\n",
        "            x = self.fc1(x)\n",
        "        x = self.relu(x)\n",
        "        if x.shape[0] > 1:\n",
        "            x = self.bn2(self.fc2(x))\n",
        "        else:\n",
        "            x = self.fc2(x)\n",
        "        x = self.relu(x)\n",
        "        x = self.fc3_mu(x)\n",
        "        return x\n",
        "\n",
        "    def decode(self, z):\n",
        "        if z.shape[0] > 1:\n",
        "            x = self.relu(self.fc_bn4(self.fc4(z)))\n",
        "            x = self.relu(self.fc_bn5(self.fc5(x))).view(-1, 64, 4, 4)\n",
        "        else:\n",
        "            x = self.relu(self.fc4(z))\n",
        "            x = self.relu(self.fc5(x)).view(-1, 64, 4, 4)\n",
        "        x = self.convTrans6(x)\n",
        "        x = self.convTrans7(x)\n",
        "        x = self.convTrans8(x)\n",
        "        x = F.interpolate(x, size=(64, 64), mode='bilinear', align_corners=True)\n",
        "        return x\n",
        "\n",
        "    def forward(self, x):\n",
        "        z = self.encode(x)\n",
        "        x_reconst = self.decode(z)\n",
        "\n",
        "        return x_reconst\n"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vrJ9bScg9AgO"
      },
      "source": [
        "# Dataset module\n",
        "\n",
        "Module for obtaining and processing data. The transform function here normalizes image's pixels from [0, 255] to [-1.0, 1.0].\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "33fWhE-h9LPq"
      },
      "source": [
        "class CustomTensorDataset(TensorDataset):\n",
        "    \"\"\"TensorDataset with support of transforms.\n",
        "    \"\"\"\n",
        "    def __init__(self, tensors):\n",
        "        self.tensors = tensors\n",
        "        if tensors.shape[-1] == 3:\n",
        "            self.tensors = tensors.permute(0, 3, 1, 2)\n",
        "        \n",
        "        self.transform = transforms.Compose([\n",
        "                            transforms.Lambda(lambda x: x.to(torch.float32)),\n",
        "                            transforms.Lambda(lambda x: 2. * x/255. - 1.),#mapping images to [-1.0, 1.0]\n",
        "                            # transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),\n",
        "                            ])\n",
        "        \n",
        "    def __getitem__(self, index):\n",
        "        x = self.tensors[index]\n",
        "        \n",
        "        if self.transform:\n",
        "            # mapping images to [-1.0, 1.0]\n",
        "            x = self.transform(x)\n",
        "\n",
        "        return x\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.tensors)"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XKNUImqUhIeq"
      },
      "source": [
        "# Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ebAJdjFmS08"
      },
      "source": [
        "## Initialize\n",
        "- hyperparameters\n",
        "- dataloader\n",
        "- model\n",
        "- optimizer & loss\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "in7yLfmqtZTk"
      },
      "source": [
        "# Training hyperparameters\n",
        "num_epochs = 50\n",
        "batch_size = 10000 # medium: smaller batchsize\n",
        "learning_rate = 1e-3\n",
        "\n",
        "# Build training dataloader\n",
        "x = torch.from_numpy(train)\n",
        "train_dataset = CustomTensorDataset(x)\n",
        "\n",
        "train_sampler = RandomSampler(train_dataset)\n",
        "train_dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=batch_size)\n",
        "\n",
        "# Model\n",
        "model_type = 'cnn'   # selecting a model type from {'cnn', 'fcn', 'vae', 'resnet'}\n",
        "model_classes = {'resnet': Resnet(), 'fcn':fcn_autoencoder(), 'cnn':conv_autoencoder(), 'vae':VAE(), }\n",
        "model = model_classes[model_type].cuda()\n",
        "\n",
        "# Loss and optimizer\n",
        "criterion = nn.MSELoss()\n",
        "optimizer = torch.optim.Adam(\n",
        "    model.parameters(), lr=learning_rate)"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wyooN-JPm8sS"
      },
      "source": [
        "## Training loop"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JoW1UrrxgI_U",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a21fd940-5322-4f14-b924-c129ecef7ab4"
      },
      "source": [
        "\n",
        "best_loss = np.inf\n",
        "model.train()\n",
        "\n",
        "qqdm_train = qqdm(range(num_epochs), desc=format_str('bold', 'Description'))\n",
        "for epoch in qqdm_train:\n",
        "    tot_loss = list()\n",
        "    for data in train_dataloader:\n",
        "\n",
        "        # ===================loading=====================\n",
        "        if model_type in ['cnn', 'vae', 'resnet']:\n",
        "            img = data.float().cuda()\n",
        "        elif model_type in ['fcn']:\n",
        "            img = data.float().cuda()\n",
        "            img = img.view(img.shape[0], -1)\n",
        "\n",
        "        # ===================forward=====================\n",
        "        output = model(img)\n",
        "        if model_type in ['vae']:\n",
        "            loss = loss_vae(output[0], img, output[1], output[2], criterion)\n",
        "        else:\n",
        "            loss = criterion(output, img)\n",
        "\n",
        "        tot_loss.append(loss.item())\n",
        "        # ===================backward====================\n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "    # ===================save_best====================\n",
        "    mean_loss = np.mean(tot_loss)\n",
        "    if mean_loss < best_loss:\n",
        "        best_loss = mean_loss\n",
        "        torch.save(model, 'best_model_{}.pt'.format(model_type))\n",
        "    # ===================log========================\n",
        "    qqdm_train.set_infos({\n",
        "      'epoch': f'{epoch + 1:.0f}/{num_epochs:.0f}',\n",
        "      'loss': f'{mean_loss:.4f}',\n",
        "    })\n",
        "    # ===================save_last========================\n",
        "    torch.save(model, 'last_model_{}.pt'.format(model_type))\n",
        "\n",
        "\n"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m                                               \n",
            " \u001b[99m0/\u001b[93m50\u001b[0m\u001b[0m   \u001b[99m        -        \u001b[0m  \u001b[99m   -    \u001b[0m                                             \n",
            "\u001b[1mDescription\u001b[0m   0.0% |                                                           |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m1/\u001b[93m50\u001b[0m\u001b[0m   \u001b[99m00:00:21<\u001b[93m00:17:53\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m1/50\u001b[0m   \u001b[99m0.3415\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m   2.0% |\u001b[97m█\u001b[0m                                                          |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m2/\u001b[93m50\u001b[0m\u001b[0m   \u001b[99m00:00:43<\u001b[93m00:17:13\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m2/50\u001b[0m   \u001b[99m0.2563\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m   4.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                                         |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m3/\u001b[93m50\u001b[0m\u001b[0m   \u001b[99m00:01:04<\u001b[93m00:16:43\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m3/50\u001b[0m   \u001b[99m0.1602\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m   6.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                                        |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m4/\u001b[93m50\u001b[0m\u001b[0m   \u001b[99m00:01:24<\u001b[93m00:16:14\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m4/50\u001b[0m   \u001b[99m0.1149\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m   8.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                                       |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m5/\u001b[93m50\u001b[0m\u001b[0m   \u001b[99m00:01:45<\u001b[93m00:15:50\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m5/50\u001b[0m   \u001b[99m0.0937\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  10.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                                      |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m6/\u001b[93m50\u001b[0m\u001b[0m   \u001b[99m00:02:06<\u001b[93m00:15:26\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m6/50\u001b[0m   \u001b[99m0.0803\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  12.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                                    |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m7/\u001b[93m50\u001b[0m\u001b[0m   \u001b[99m00:02:27<\u001b[93m00:15:03\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m7/50\u001b[0m   \u001b[99m0.0731\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  14.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                                   |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m8/\u001b[93m50\u001b[0m\u001b[0m   \u001b[99m00:02:47<\u001b[93m00:14:40\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m8/50\u001b[0m   \u001b[99m0.0614\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  16.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                                  |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m9/\u001b[93m50\u001b[0m\u001b[0m   \u001b[99m00:03:08<\u001b[93m00:14:18\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m9/50\u001b[0m   \u001b[99m0.0536\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  18.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                                 |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m10/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:03:29<\u001b[93m00:13:57\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m10/50\u001b[0m  \u001b[99m0.0473\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  20.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                                |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m11/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:03:50<\u001b[93m00:13:36\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m11/50\u001b[0m  \u001b[99m0.0472\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  22.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                               |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m12/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:04:11<\u001b[93m00:13:15\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m12/50\u001b[0m  \u001b[99m0.0455\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  24.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                             |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m13/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:04:31<\u001b[93m00:12:53\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m13/50\u001b[0m  \u001b[99m0.0406\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  26.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                            |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m14/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:04:52<\u001b[93m00:12:31\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m14/50\u001b[0m  \u001b[99m0.0382\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  28.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                           |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m15/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:05:13<\u001b[93m00:12:10\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m15/50\u001b[0m  \u001b[99m0.0362\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  30.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                          |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m16/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:05:33<\u001b[93m00:11:49\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m16/50\u001b[0m  \u001b[99m0.0351\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  32.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                         |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m17/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:05:54<\u001b[93m00:11:28\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m17/50\u001b[0m  \u001b[99m0.0342\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  34.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                       |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m18/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:06:15<\u001b[93m00:11:06\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m18/50\u001b[0m  \u001b[99m0.0340\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  36.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                      |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m19/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:06:35<\u001b[93m00:10:45\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m19/50\u001b[0m  \u001b[99m0.0323\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  38.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                     |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m20/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:06:56<\u001b[93m00:10:25\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m20/50\u001b[0m  \u001b[99m0.0310\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  40.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                    |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m21/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:07:17<\u001b[93m00:10:03\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m21/50\u001b[0m  \u001b[99m0.0300\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  42.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                   |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m22/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:07:38<\u001b[93m00:09:42\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m22/50\u001b[0m  \u001b[99m0.0302\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  44.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                  |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m23/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:07:58<\u001b[93m00:09:21\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m23/50\u001b[0m  \u001b[99m0.0294\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  46.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                                |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m24/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:08:19<\u001b[93m00:09:00\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m24/50\u001b[0m  \u001b[99m0.0287\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  48.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                               |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m25/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:08:39<\u001b[93m00:08:39\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m25/50\u001b[0m  \u001b[99m0.0272\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  50.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                              |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m26/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:08:59<\u001b[93m00:08:18\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m26/50\u001b[0m  \u001b[99m0.0276\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  52.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                             |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m27/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:09:20<\u001b[93m00:07:57\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m27/50\u001b[0m  \u001b[99m0.0273\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  54.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                            |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m28/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:09:41<\u001b[93m00:07:36\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m28/50\u001b[0m  \u001b[99m0.0259\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  56.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                          |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m29/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:10:01<\u001b[93m00:07:15\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m29/50\u001b[0m  \u001b[99m0.0267\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  58.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                         |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m30/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:10:21<\u001b[93m00:06:54\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m30/50\u001b[0m  \u001b[99m0.0266\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  60.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                        |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m31/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:10:42<\u001b[93m00:06:33\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m31/50\u001b[0m  \u001b[99m0.0253\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  62.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                       |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m32/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:11:02<\u001b[93m00:06:12\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m32/50\u001b[0m  \u001b[99m0.0252\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  64.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                      |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m33/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:11:23<\u001b[93m00:05:51\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m33/50\u001b[0m  \u001b[99m0.0253\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  66.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                     |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m34/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:11:43<\u001b[93m00:05:31\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m34/50\u001b[0m  \u001b[99m0.0248\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  68.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                   |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m35/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:12:03<\u001b[93m00:05:10\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m35/50\u001b[0m  \u001b[99m0.0246\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  70.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                  |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m36/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:12:24<\u001b[93m00:04:49\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m36/50\u001b[0m  \u001b[99m0.0237\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  72.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                 |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m37/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:12:44<\u001b[93m00:04:28\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m37/50\u001b[0m  \u001b[99m0.0232\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  74.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m                |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m38/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:13:05<\u001b[93m00:04:08\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m38/50\u001b[0m  \u001b[99m0.0229\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  76.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m               |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m39/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:13:25<\u001b[93m00:03:47\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m39/50\u001b[0m  \u001b[99m0.0229\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  78.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m             |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m40/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:13:46<\u001b[93m00:03:26\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m40/50\u001b[0m  \u001b[99m0.0219\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  80.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m            |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m41/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:14:06<\u001b[93m00:03:05\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m41/50\u001b[0m  \u001b[99m0.0219\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  82.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m           |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m42/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:14:27<\u001b[93m00:02:45\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m42/50\u001b[0m  \u001b[99m0.0226\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  84.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m          |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m43/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:14:47<\u001b[93m00:02:24\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m43/50\u001b[0m  \u001b[99m0.0225\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  86.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m         |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m44/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:15:08<\u001b[93m00:02:03\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m44/50\u001b[0m  \u001b[99m0.0272\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  88.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m        |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m45/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:15:28<\u001b[93m00:01:43\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m45/50\u001b[0m  \u001b[99m0.0230\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  90.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m      |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m46/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:15:49<\u001b[93m00:01:22\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m46/50\u001b[0m  \u001b[99m0.0218\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  92.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m     |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m47/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:16:09<\u001b[93m00:01:01\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m47/50\u001b[0m  \u001b[99m0.0204\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  94.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m    |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m48/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:16:29<\u001b[93m00:00:41\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m48/50\u001b[0m  \u001b[99m0.0222\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  96.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m   |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m49/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:16:50<\u001b[93m00:00:20\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m49/50\u001b[0m  \u001b[99m0.0198\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m  98.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m  |\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m50/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:17:10<\u001b[93m00:00:00\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m50/50\u001b[0m  \u001b[99m0.0200\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m 100.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m|\u001b[K\u001b[F\u001b[K\u001b[F \u001b[1mIters\u001b[0m    \u001b[1mElapsed Time\u001b[0m      \u001b[1mSpeed\u001b[0m    \u001b[1mepoch\u001b[0m   \u001b[1mloss\u001b[0m                               \n",
            " \u001b[99m50/\u001b[93m50\u001b[0m\u001b[0m  \u001b[99m00:17:10<\u001b[93m00:00:00\u001b[0m\u001b[0m  \u001b[99m0.05it/s\u001b[0m  \u001b[99m50/50\u001b[0m  \u001b[99m0.0200\u001b[0m                              \n",
            "\u001b[1mDescription\u001b[0m 100.0% |\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m\u001b[97m█\u001b[0m|"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wk0UxFuchLzR"
      },
      "source": [
        "# Inference\n",
        "Model is loaded and generates its anomaly score predictions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "evgMW3OwoGqD"
      },
      "source": [
        "## Initialize\n",
        "- dataloader\n",
        "- model\n",
        "- prediction file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_MBnXAswoKmq"
      },
      "source": [
        "eval_batch_size = 200\n",
        "\n",
        "# build testing dataloader\n",
        "data = torch.tensor(test, dtype=torch.float32)\n",
        "test_dataset = CustomTensorDataset(data)\n",
        "test_sampler = SequentialSampler(test_dataset)\n",
        "test_dataloader = DataLoader(test_dataset, sampler=test_sampler, batch_size=eval_batch_size, num_workers=1)\n",
        "eval_loss = nn.MSELoss(reduction='none')\n",
        "\n",
        "# load trained model\n",
        "checkpoint_path = 'best_model_cnn.pt'\n",
        "model = torch.load(checkpoint_path)\n",
        "model.eval()\n",
        "\n",
        "# prediction file \n",
        "out_file = 'PREDICTION_FILE.csv'"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r1PS_ApzhfOQ"
      },
      "source": [
        "    \n",
        "anomality = list()\n",
        "with torch.no_grad():\n",
        "  for i, data in enumerate(test_dataloader): \n",
        "        if model_type in ['cnn', 'vae', 'resnet']:\n",
        "            img = data.float().cuda()\n",
        "        elif model_type in ['fcn']:\n",
        "            img = data.float().cuda()\n",
        "            img = img.view(img.shape[0], -1)\n",
        "        else:\n",
        "            img = data[0].cuda()\n",
        "        output = model(img)\n",
        "        if model_type in ['cnn', 'resnet', 'fcn']:\n",
        "            output = output\n",
        "        elif model_type in ['res_vae']:\n",
        "            output = output[0]\n",
        "        elif model_type in ['vae']: # , 'vqvae'\n",
        "            output = output[0]\n",
        "        if model_type in ['fcn']:\n",
        "            loss = eval_loss(output, img).sum(-1)\n",
        "        else:\n",
        "            loss = eval_loss(output, img).sum([1, 2, 3])\n",
        "        anomality.append(loss)\n",
        "anomality = torch.cat(anomality, axis=0)\n",
        "anomality = torch.sqrt(anomality).reshape(len(test), 1).cpu().numpy()\n",
        "\n",
        "df = pd.DataFrame(anomality, columns=['Predicted'])\n",
        "df.to_csv(out_file, index_label = 'Id')\n",
        "\n"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kXUP4iWFQdY8",
        "outputId": "33053720-51be-420f-9f18-1790f12c0bef"
      },
      "source": [
        "!ls -l"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "total 26673\n",
            "-rw------- 1 root root   788213 Aug 20 04:03 best_model_cnn.pt\n",
            "-rw------- 1 root root 12713077 Aug 20 03:35 best_model_fcn.pt\n",
            "drwx------ 2 root root     4096 Aug 18 02:33 data-bin\n",
            "-rw------- 1 root root   788213 Aug 20 04:04 last_model_cnn.pt\n",
            "-rw------- 1 root root 12713077 Aug 20 03:35 last_model_fcn.pt\n",
            "-rw------- 1 root root   305118 Aug 20 04:11 PREDICTION_FILE.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ATBy-JhH7TUt"
      },
      "source": [
        "# Training statistics\n",
        "- Number of parameters\n",
        "- Training time on colab\n",
        "- Training curve of the bossbaseline model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HxVv7IYfuTHE"
      },
      "source": [
        "- Simple\n",
        " - Number of parameters: 3176419\n",
        " - Training time on colab: ~ 30 min\n",
        "- Medium\n",
        " - Number of parameters: 47355\n",
        " - Training time on colab: ~ 30 min\n",
        "- Strong\n",
        " - Number of parameters: 47595\n",
        " - Training time on colab:  4 ~ 5 hrs\n",
        "- Boss:  \n",
        " - Number of parameters: 4364140\n",
        " - Training time on colab: 1.5~3 hrs\n",
        "\n",
        " ![Screen Shot 2021-04-29 at 16.43.57.png]()\n"
      ]
    }
  ]
}